Performance of Scheduled Tasks via Behavior Analysis and Dynamic Optimization

ABSTRACT

In a method for improving escalation performance, a computer monitors performance of an escalation in a production environment. The computer identifies a characteristic of the escalation based on the monitored performance. The computer creates a recommendation for improving escalation performance based on the characteristic. In response to an approval of the recommendation, the computer applies the recommendation to the escalation to form one or more recommended escalations. Furthermore, the computer deploys the one or more recommended escalations into the production environment.

TECHNICAL FIELD

The present application relates generally to an improved data processingapparatus and method and more specifically to mechanisms for improvingthe performance of scheduled tasks via behavior analysis and dynamicoptimization.

BACKGROUND

An escalation application provides a way to schedule tasks withconfigurable conditions to trigger actions and/or send notifications.The scheduled tasks with their associated configurable conditions,actions, and notifications are referred to herein as “escalations.” Thatis, an escalation is a non-system scheduled task with triggeringconditions that resides within an application domain. By “non-system”what is meant is that an escalation runs within the application, domain,or web container with which the escalation is implement. An escalationcan be reoccurring and enables automatic execution of actions ornotifications when associated criteria is met, leveraging configurationdata and security models defined in the application. An escalation canhave escalation points, which are essentially sub-conditions that may becustomizable to the particular implementer of the escalation. Thus, theescalation is a “super-condition” that must be met before the“sub-conditions” or escalation points, are evaluated.

Escalations may be scheduled to automatically run at specified timeintervals, discrete time points, or the like. The escalations may becustomized or tailored to a particular user's needs via the escalationapplication. An example of an escalation application may be, forexample, an escalation application that monitors processes to make surethat critical processes are performed in a desired time period. Thus, anescalation may be a task that executes periodically to monitor theexecution of processes, determine whether their execution times meet oneor more criteria, and then perform one or more actions based on the oneor more criteria being met or not, e.g., if the average execution timeof a process is greater than a predetermined threshold, the escalationmay increase a severity of a condition, issue a trouble ticket, sendnotifications to appropriate individuals, or perform other actions.

SUMMARY

In one illustrative embodiment, a method is provided for improvingescalation performance. The method comprises a computer monitoringperformance of an escalation in a production environment. The methodfurther comprises the computer identifying a characteristic of theescalation based on the monitored performance. The method also comprisesthe computer creating a recommendation for improving escalationperformance based on the characteristic. In addition, the methodcomprises in response to an approval of the recommendation, the computerapplying the recommendation to the escalation to form one or morerecommended escalations. Furthermore, the method comprises the computerdeploying the one or more recommended escalations into the productionenvironment.

In another illustrative embodiment, a system is provided for improvingescalation performance. The system comprises one or more processors, oneor more computer-readable memories, and one or more computer-readabletangible storage devices. The system further comprises programinstructions, stored on at least one of the one or morecomputer-readable tangible storage devices for execution by at least oneof the one or more processors via at least one of the one or morecomputer-readable memories, to monitor performance of an escalation in aproduction environment. Moreover, the system comprises programinstructions, stored on at least one of the one or morecomputer-readable tangible storage devices for execution by at least oneof the one or more processors via at least one of the one or morecomputer-readable memories, to identify a characteristic of theescalation based on the monitored performance. Furthermore, the systemcomprises program instructions, stored on at least one of the one ormore computer-readable tangible storage devices for execution by atleast one of the one or more processors via at least one of the one ormore computer-readable memories, to create a recommendation forimproving escalation performance based on the characteristic. Inaddition, the system comprises program instructions, stored on at leastone of the one or more computer-readable tangible storage devices forexecution by at least one of the one or more processors via at least oneof the one or more computer-readable memories, to apply therecommendation to the escalation to form one or more recommendedescalations in response to an approval of the recommendation. The systemalso comprises program instructions, stored on at least one of the oneor more computer-readable tangible storage devices for execution by atleast one of the one or more processors via at least one of the one ormore computer-readable memories, to deploy the one or more recommendedescalations into the production environment.

In yet another illustrative embodiment, a computer program product forimproving escalation performance is provided. The computer programproduct comprises one or more computer-readable tangible storage devicesand program instructions, stored on at least one of the one or morecomputer-readable tangible storage devices, to monitor performance of anescalation in a production environment. The computer program productfurther comprise's program instructions, stored on at least one of theone or more computer-readable tangible storage devices, to identify acharacteristic of the escalation based on the monitored performance. Thecomputer program product also comprises program instructions, stored onat least one of the one or more computer-readable tangible storagedevices, to create a recommendation for improving future escalationperformance based on the characteristic. In addition, the computerprogram product comprises program instructions, stored on at least oneof the one or more computer-readable tangible storage devices, to applythe recommendation to the escalation to form one or more recommendedescalations in response to an approval of the recommendation. Moreover,the computer program product comprises program instructions, stored onat least one of the one or more computer-readable tangible storagedevices, to deploy the one or more recommended escalations into theproduction environment.

These and other features and advantages of the present invention will bedescribed in, or will become apparent to those of ordinary skill in theart in view of, the following detailed description of the exampleembodiments of the present invention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The invention, as well as a preferred mode of use and further objectivesand advantages thereof, will best be understood by reference to thefollowing detailed description of illustrative embodiments when read inconjunction with the accompanying drawings, wherein:

FIG. 1 is a pictorial representation of an example distributed dataprocessing system in which aspects of the illustrative embodiments maybe implemented;

FIG. 2 is a block diagram of an example data processing system in whichaspects of the illustrative embodiments may be implemented;

FIGS. 3A and 3B provide an example diagram illustrating a graphical userinterface that may be provided by an escalation application for definingand/or modifying an escalation in accordance with one illustrativeembodiment;

FIG. 4 is an example block diagram of a proactive monitoring and dynamicoptimization engine in accordance with one illustrative embodiment;

FIG. 5 is an example diagram of one recommended action list entry thatmay be used with this recommended action list data structure inaccordance with one illustrative embodiment;

FIG. 6 is a flowchart outlining an example operation for performing areal-time behavior analysis of an escalation in accordance with oneillustrative embodiment;

FIG. 7 is a flowchart outlining an example operation for performing ahistorical trend behavior analysis of an escalation in accordance withone illustrative embodiment;

FIG. 8 is a flowchart outlining an example operation for performing hungescalation analysis in accordance with one illustrative embodiment; and

FIG. 9 is a flowchart outlining an example operation for performing anapproval/commit operation in accordance with one illustrativeembodiment.

DETAILED DESCRIPTION

There are times when scheduled tasks defined in an escalationapplication may not execute properly and may trigger an action, such asincreasing a severity of an incident and issuing an incident ticket, dueto an environment issue, heavy load on the system, or the like. Forexample, there are times when an escalation may hang due to varioussituations, e.g., an escalation that sends out a notification byconnecting to an SMTP server may hang if the SMTP server is down or notresponding, an escalation that issues a query to a database may hang ifthe database does not respond or if the query is complex and thedatabase takes longer than an acceptable amount of time to respond, anescalation that executes but does not finish or does not process all ofthe required records, or the like. Illustrative embodiments recognizethat scheduled tasks defined in an escalation application and notexecuting properly may cause an interruption in a user's businessoperations. Illustrative embodiments further recognize that, in a sharedenvironment, scheduled tasks defined in an escalation application andnot executing properly may impact all customers residing on the sharedinstance. Illustrative embodiments further recognize that these issuesmay be highly visible to upper management and typically require largeproblem determination efforts in order to resolve, often involvingsupport teams, implementation/transition teams, system administrators,subject matter experts, and the like.

Currently, these issues are handled manually. That is, a systemadministrator or the like must manually verify the escalations areworking properly. Illustrative embodiments recognize that this is quitetime consuming, is reactive in nature, and may introduce human error.

The illustrative embodiments improve performance of scheduled tasks viabehavior analysis and dynamic optimization. In the illustrativeembodiments described herein, the scheduled tasks are escalationsdefined and managed via an escalation application although the inventionis not limited to such and may be applied to any scheduled tasks whetherthey are escalations or other types of scheduled tasks. As mentionedabove, escalations are non-system scheduled tasks with triggeringconditions that reside within an application domain, can be reoccurring,and enable automatic execution of actions and/or notifications when thecriteria of the escalation is met, leveraging configuration data andsecurity models defined in the corresponding application with which theescalations are implemented. The illustrative embodiments proactivelymonitor and correct escalations. For example, the illustrativeembodiments attempt to prevent failure or hanging of escalations byanalyzing the behavior of the escalations and adjusting variousparameters that will reduce the likelihood of escalation executionfailure. If an escalation does fail, however, the illustrativeembodiments detect such failure and provide remediation operations sothat the escalation will start executing successfully again.

The illustrative embodiments provide an automated functionality forproactively monitoring escalations and dynamically optimize theescalation application. The dynamic and automatic monitoring/alertcreation and automatic parameter adjustment features of the illustrativeembodiments prevent high severity issues from being erroneously createdand greatly enhance the overall escalation application performance. Theillustrative embodiments further minimize or eliminate the manual effortinvolved in verification and modification of escalations, thereby alsominimizing or eliminating human error.

Illustrative embodiments provide for performing behavior analysis on acollection of scheduled tasks, which for purposes of this descriptionare assumed to be escalations of an escalation application. Based onthis behavior analysis, a determination may be made as to whether animprovement may be made to the execution of the escalation. In responseto such a determination, the improvement to the execution of theescalation may be automatically implemented using the illustrativeembodiments, or a system administrator may be automatically prompted viaan automatically generated notification to implement the identifiedimprovement. Implementation of the identified improvement may involveproviding the user with one or more options, e.g., via a menu or otheruser interface, for selecting whether to automatically apply theimprovement immediately, automatically apply the improvement at a nextexecution of the escalation, or to schedule the automatic application ofthe improvement during a planned change window, i.e. a future time whenchanges to escalations may be made without significantly degrading theperformance of the computing system.

In one illustrative embodiment, the automatic determination of animprovement to the execution of the escalation may include automaticallydetecting “hung” escalations, where a “hung” (non-responsive, stuck,failed, or the like) escalation is an escalation that fails to complete,fails to complete correctly by failing to process all data/records thatthe escalation was to process, or completes beyond an allottedprocessing time. For example, if an escalation is scheduled to run at anpredetermined time interval N, then a hung escalation may be anescalation in which the current time is greater than the sum of a lastrun time+N interval+delay value, where the delay value is apre-determined value that is an estimate of how long it will take beforean escalation will be detected as being “hung”. The delay value may bepre-determined, may be calculated based on previous results of theexecution of the escalation, historical analysis, e.g., averageexecution times, or the like.

In such a case, actions may be identified for reviving the hungescalation before the hung escalation causes a greater impact onapplication execution. For example, the hung escalation may be disabled,a new escalation similar to the hung escalation may be dynamicallycreated that has similar execution criteria as the hung escalation, butmodified to execute only on data or records that have not yet beenprocessed by the hung escalation. This new escalation may then beenabled and activated such that the new escalation takes the place ofthe hung escalation, and such that the hung escalation is disabled andno longer used. In order to avoid missing data, the new escalation'scondition is defined so that it starts from a last successful runtimeand reprocesses the already processed records/data of the hungescalation as well as those records/data that were not processed by thehung escalation. In such a case, the already processed records/data maybe automatically skipped by the new escalation in response to detectingthat these records/data were already processed by the hung escalation,e.g., if the first 500 records were processed by the hung escalation,then the first 500 records would be skipped by the new escalation andthe new escalation would begin processing at record 501.

In other illustrative embodiments, the behavior analysis may involveconstructing a historical trend of monitored escalations. The historicaltrend of the monitored escalations may include such parameters as timetaken to execute the escalation, amount of data or number of recordsaffected by escalation execution, system load at the time that theescalation executes, and the like. The historical trend may be analyzedand used to define a baseline for evaluation; of escalation executions.The execution parameters of an escalation may be compared against thesebaselines to determine if modification to the escalation should beperformed so as to improve the execution of the escalation and reducethe probability that the escalation will fail. These modifications mayinclude dynamically adjusting a delay parameter associated with theescalation, dynamically adjusting an interval for execution of theescalation so as to stagger execution times of escalations, dynamicallymodifying the execution server of the escalation, dynamically modifyingthe execution condition criteria of the escalation, splitting anescalation into two or more escalations with various execution criteriathat together comprise the criteria of the original escalation, and thelike. Each of these illustrative embodiments will be described ingreater detail hereafter with reference to the figures.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method, or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in any one or more computer readablemedium(s) having computer usable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CDROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, in abaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, radio frequency (RF), etc., or anysuitable combination of the foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java™, Smalltalk™, C++, or the like, and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. Java and all Java-based trademarks and logos aretrademarks or registered trademarks of Oracle and/or its affiliates. Theprogram code may execute entirely on the user's computer, partly on theuser's computer, as a stand-alone software package, partly on the user'scomputer and partly on a remote computer, or entirely on the remotecomputer or server. In the latter scenario, the remote computer may beconnected to the user's computer through any type of network, includinga local area network (LAN) or a wide area network (WAN), or theconnection may be made to an external computer (for example, through theInternet using an Internet Service Provider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to the illustrativeembodiments of the invention. It will be understood that each block ofthe flowchart illustrations and/or block diagrams, and combinations ofblocks in the flowchart illustrations and/or block diagrams, can beimplemented by computer program instructions. These computer programinstructions may be provided to a processor of a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions thatimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus, or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

Thus, the illustrative embodiments may be utilized in many differenttypes of data processing environments. In order to provide a context forthe description of the specific elements and functionality of theillustrative embodiments, FIGS. 1 and 2 are provided hereafter asexample environments in which aspects of the illustrative embodimentsmay be implemented. It should be appreciated that FIGS. 1 and 2 are onlyexamples and are not intended to assert or imply any limitation withregard to the environments in which aspects or embodiments of thepresent invention may be implemented. Many modifications to the depictedenvironments may be made without departing from the spirit and scope ofthe present invention.

FIG. 1 depicts a pictorial representation of an example distributed dataprocessing system in which aspects of the illustrative embodiments maybe implemented. Distributed data processing system 100 may include anetwork of computers in which aspects of the illustrative embodimentsmay be implemented. The distributed data processing system 100 containsat least one network 102, which is the medium used to providecommunication links between various devices and computers connectedtogether within distributed data processing system 100. The network 102may include connections, such as wire, wireless communication links, orfiber optic cables.

In the depicted example, server 104 and server 106 are connected tonetwork 102 along with storage unit 108. In addition, clients 110, 112,and 114 are also connected to network 102. These clients 110, 112, and114 may be, for example, personal computers, network computers, or thelike. In the depicted example, server 104 provides data, such as bootfiles, operating system images, and applications to the clients 110,112, and 114. Clients 110, 112, and 114 are clients to server 104 in thedepicted example. Distributed data processing system 100 may includeadditional servers, clients, and other devices not shown.

In the depicted example, distributed data processing system 100 is theInternet with network 102 representing a worldwide collection ofnetworks and gateways that use the Transmission ControlProtocol/Internet Protocol (TCP/IP) suite of protocols to communicatewith one another. At the heart of the Internet is a backbone ofhigh-speed data communication lines between major nodes or hostcomputers, consisting of thousands of commercial, governmental,educational and other computer systems that route data and messages. Ofcourse, the distributed data processing system 100 may also beimplemented to include a number of different types of networks, such asfor example, an intranet, a local area network (LAN), a wide areanetwork (WAN), or the like. As stated above, FIG. 1 is intended as anexample, not as an architectural limitation for different embodiments ofthe present invention, and therefore, the particular elements shown inFIG. 1 should not be considered limiting with regard to the environmentsin which the illustrative embodiments of the present invention may beimplemented.

FIG. 2 is a block diagram of an example data processing system in whichaspects of the illustrative embodiments may be implemented. Dataprocessing system 200 is an example of a computer, such as client 110 orserver 104 in FIG. 1, in which computer usable code or instructionsimplementing the processes for illustrative embodiments of the presentinvention may be located.

In the depicted example, data processing system 200 employs a hubarchitecture including north bridge and memory controller hub (NB/MCH)202 and south bridge and input/output (I/O) controller hub (SB/ICH) 204.Processing Unit 206, main memory 208, and graphics processor 210 areconnected to NB/MCH 202. Processing unit 206 may include one or moreprocessors and may be implemented using one or more heterogeneousprocessor systems. Graphics processor 210 may be connected to NB/MCH 202through an accelerated graphics port (AGP).

In the depicted example, local area network (LAN) adapter 212 connectsto SB/ICH 204. Audio adapter 216, keyboard and mouse adapter 220, modem222, read only memory (ROM) 224, universal serial bus (USB) ports andother communication ports 232, and PCI/PCIe devices 234 connect toSB/ICH 204 through/bus 238. PCI/PCIe devices 234 may include, forexample, Ethernet adapters, add-in cards, and PC cards for notebookcomputers. PCI uses a card bus controller, while PCIe does not. ROM 224may be, for example, a flash basic input/output system (BIOS).

Hard disk drive (HDD) 226 and CD-ROM drive 230 connect to SB/ICH 204through bus 240. HDD 226 and CD-ROM drive 230 may use, for example, anintegrated drive electronics (IDE) or serial advanced technologyattachment (SATA) interface. Super I/O (SIO) device 236 may be connectedto SB/ICH 204 through bus 238.

An operating system runs on processing unit 206. The operating systemcoordinates and provides control of various components within the dataprocessing system 200 in FIG. 2. As a client, the operating system maybe a commercially available operating system such as Microsoft® Windows7®. Microsoft, Windows, Windows NT, and the Windows logo are trademarksof Microsoft Corporation in the United States, other countries, or both.An object-oriented programming system, such as the Java™ programmingsystem, may run in conjunction with the operating system and providescalls to the operating system from Java™ programs or applicationsexecuting on data processing system 200.

As a server, data processing system 200 may be, for example, an IBM®eServer™ System p® computer system, running the AdvancedInteractive-Executive (AIX®) operating system or the LINUX operatingsystem. Linux is a registered trademark of Linus Torvalds in the UnitedStates, other countries, or both. Data processing system 200 may be asymmetric multiprocessor (SMP) system including a plurality ofprocessors in processing unit 206. Alternatively, a single processorsystem may be employed.

Instructions for the operating system, the object-oriented programmingsystem, and applications or programs are located on storage devices,such as HDD 226, and may be loaded into main memory 208 for execution byprocessing unit 206. The processes for illustrative embodiments of thepresent invention may be performed by processing unit 206 executingcomputer usable program code, which may be loaded into a memory such as,for example, main memory 208, ROM 224, and which may be stored on one ormore storage devices, such as HDD 226 and CD-ROM drive 230.

A bus system, such as bus 238 or bus 240 as shown in FIG. 2, may becomprised of one or more buses. Of course, the bus system may beimplemented using any type of communication fabric or architecture thatprovides for a transfer of data between different components or devicesattached to the fabric or architecture. A communication unit, such asmodem 222 or network adapter 212 of FIG. 2, may include one or moredevices used to transmit and receive data. A memory may be for example,main memory 208, ROM 224, or a cache such as found in NB/MCH 202 in FIG.2.

Those of ordinary skill in the art will appreciate that the hardware inFIGS. 1 and 2 may vary depending on the implementation. Other internalhardware or peripheral devices, such as flash memory, equivalentnon-volatile memory, or optical disk drives and the like, may be used inaddition to or in place of the hardware depicted in FIGS. 1 and 2. Also,the processes of the illustrative embodiments may be applied to amultiprocessor data processing system, or an SMP system, withoutdeparting from the spirit and scope of the present invention.

Moreover, the data processing system 200 may take the form of any of anumber of different data processing systems including client computingdevices, server computing devices, a tablet computer, laptop computer,telephone or other communication device, a personal digital assistant(PDA), or the like. In some illustrative examples, data processingsystem 200 may be a portable computing device that is configured withflash memory to provide non-volatile memory for storing operating systemfiles and/or user-generated data, for example. Essentially, dataprocessing system 200 may be any known or later developed dataprocessing system without architectural limitation.

Referring again to FIG. 1, one or more of the computing devices in FIG.1, e.g., server 104 and/or 106 may implement a service management devicethat may execute one or more service management applications forinformation technology service management. Such service managementapplications may include applications such as self-help for end users,full service desk incident, problem and change (IPC) managementapplications with service support, asset management, and the like. Anexample of the one or more service management applications with whichthe illustrative embodiments may be operate is the IBM ServiceManagement (ISM) software suite available from International BusinessMachines Corporation of Armonk, N.Y. A service management suite is a setof applications and data structures that permit system administratorsand other authorized personnel to monitor and manage the variousservices, business systems, data storage systems, and the like, of anorganization and often provide a plurality of graphical user interfacesfor facilitating such monitoring and management.

The incident, problem and change management applications of the servicemanagement system may implement one or more scheduled tasks and mayprovide an application for defining and managing these scheduled tasks.For example, in the illustrative embodiments described herein, these oneor more scheduled tasks are escalations, i.e. a particular type ofscheduled task as defined above, and the incident, problem and changemanagement applications may implement an escalation application fordefining and managing such escalations. It should be appreciated thatwhile the illustrative embodiments will be described in terms ofescalations and escalation applications, the illustrative embodimentsare not limited to such and in other implementations may on other typesof scheduled tasks other than escalations.

FIGS. 3A and 3B provide an example diagram illustrating a graphical userinterface that may be provided by an escalation application for definingand/or modifying an escalation in accordance with one illustrativeembodiment. The escalation being defined using the graphical userinterface 300 of FIG. 3, in this example, is an escalation whose purposeis to change the status of expired labor contracts from an active statusto an expired status. Many different types of escalations can be usedwithout departing from the spirit and scope of the illustrativeembodiments.

As shown in FIGS. 3A and 3B, the graphical user interface 300 comprisesa first portion 310 for defining the name of the escalation 312, thetarget of the escalation 314. e.g., a target database, field of adatabase, or the like, a condition checked by the escalation 316, andthe like. A second portion 320 of the graphical user interface 300 isprovided for specifying escalation points and their elapsed timeattributes. The escalation points are customer specific conditions thatare a subset of the general escalation condition 316. A third portion330 is provided for defining actions to be performed if the condition ofthe escalation 316 is met. In the depicted example, if the condition 316is met, then the action “LABCTREXPD” 332, which is a change status typeof action, is executed. This action sets an expired labor contract toEXPRD status. A fourth portion 340 is provided for defining the scheduleof the escalation execution. In the depicted example, the escalation isexecuted every 24 hours and a preview of the execution times is shown inthe fourth portion 340.

It should be appreciated that the graphical user interface 300 is onlyone example of a means for defining and modifying escalations. Othermeans for establishing escalations may be used without departing fromthe spirit and scope of the illustrative embodiments.

In accordance with the illustrative embodiments, as a further part ofthe incident, problem, and change management applications of the servicemanagement system, the illustrative embodiments provide a proactivemonitoring and dynamic optimization engine. The proactive monitoring anddynamic optimization engine augments and enhances the capabilities of aservice management system and its corresponding one or more servicemanagement applications by providing the various functionality describedhereafter. The proactive monitoring and dynamic optimization engine maybe implemented as software code stored on one or more storage devicesand executed on one or more processors using associated hardwareincluding one or more memories, buses, co-processors, or the like. Inother illustrative embodiments, the proactive monitoring and dynamicoptimization engine may be implemented in hardware or at least partiallyin hardware, such as firmware, application specific integrated circuits(ASICs), or the like. For purposes of the present description, it willbe assumed that the proactive monitoring and dynamic optimization engineis implemented as program instructions, which may be stored in one ormore computer-readable storage devices (e.g., HDD 226 or CD-ROM drive230 of FIG. 2), and which may be executed by one or more processors(e.g., processing unit 206 of FIG. 2) of one or more data processingdevices via one or more computer-readable memories (e.g., main memory208 of FIG. 2).

The proactive monitoring and dynamic optimization engine includes aplurality of components for performing various operations includingbehavior analysis, load measurements, generating recommendations,performing simulations, updating escalations, staggering escalationexecution times, detecting hung escalations, performing modifications tohung escalations, and the like. Each of these will be described indetail hereafter. The primary purpose of the proactive monitoring anddynamic optimization engine is to provide an automated facility forproactively monitoring escalations and dynamically optimizing theescalation application and its escalations.

FIG. 4 is an example block diagram of a proactive monitoring and dynamicoptimization engine in accordance with one illustrative embodiment. Thevarious elements of the proactive monitoring and dynamic optimizationengine each have logic within them for performing the various operationsattributed to them in the description hereafter. This logic may beprovided as software logic (program instructions), hardware logic, orany combination of software and hardware logic. For purposes of thisdescription, it will be assumed that the elements of the proactivemonitoring and dynamic optimization engine are implemented as programinstructions that are stored in one or more computer-readable storagedevices (e.g., HDD 226 or CD-ROM drive 230 of FIG. 2 and that areexecuted by one or more processors (e.g., processing unit 206 of FIG. 2)of one or more data processing devices via one or more computer-readablememories (e.g., main memory 208 of FIG. 2).

As shown in FIG. 4, the proactive monitoring and dynamic optimizationengine 400 includes a behavior analysis module 410, an escalation metricmeasurement module 420, are commendation module 430, anapproval/confirmation module 440, a simulation module 450, an escalationupdate module 460, a hung escalation detector 470, an alert/notificationmodule 480, an escalation fix module 490, and an escalation staggeringmodule 495. The proactive monitoring and dynamic optimization engine 400interacts with and works in conjunction with escalation application 498.Control logic 405 is provided in the proactive monitoring and dynamicoptimization engine 400 for controlling the overall operation of theproactive monitoring and dynamic optimization engine 400 andorchestrating the operation of the other elements 410-495 of theproactive monitoring and dynamic optimization engine 400.

The behavior analysis module 410 determines whether dynamic optimizationand performance improvement is warranted for one or more escalationsidentified in a monitored escalation list data structure 416 thatidentifies the particular escalations 497 defined by and managed by theescalation application 498 that are to be analyzed for purposes ofdetermining whether dynamic optimization and performance improvementsare to be performed. The particular escalations 497 that are in themonitored escalation list data structure 416 may be identified orselected from the escalation application 498 by a system administratoror other authorized user. The behavior analysis module 410 performsevaluations on behavior information obtained from the execution of thesevarious escalations and determines a list of recommended actions to beperformed to optimize and improve performance of these escalations. Thebehavior information may be obtained, by the escalation metricmeasurement module 420, from the execution of the escalations in aproduction environment.

The escalation metric measurement module 420 may collect executionmetrics for escalations, which may include all of the escalations 497defined and managed by the escalation application 498 or only thoseescalations in the monitored escalation list data structure 416. Thesecollected metrics are used as a basis for performing the behavioranalysis by the behavior analysis module 410. Such metrics may include anumber of transactions evaluated by an escalation, execution times of anescalation, for which users and/or client applications an escalation isexecuted, etc. These metrics may be obtained, for example, from anescalation log data structure 499 maintained by the escalationapplication 498. That is, the escalation application 498 may monitor theexecution of escalations 497 and store, in the escalation log datastructure 499, information regarding the times when escalations 497 areexecuted, for which users/client applications the escalations 497 areexecuted, how long the executions of the escalations 497 took tocomplete, how many times the escalations 497 are executed for eachuser/client application, a last run time of the escalations 497 for eachuser/client application, and other log information that may be laterused by the behavior analysis module 410 to perform behavior analysis ofthe various escalations 497. While FIG. 4 shows the escalation log datastructure 499 as a separate data structure, in other illustrativeembodiments, the escalation log data structure 499 may be incorporatedinto a database that is the target of at least one of the escalations497. It should be further appreciated that any configurable parameter ofat least one of the escalations 497, as well as any system parameters,e.g., CPU, memory, etc., may be used by the behavior analysis module 410to perform its operations as described herein.

The behavior analysis module 410 performs both real-time and historicalanalysis on the measured escalation metrics to determine whether todynamically optimize or improve the performance of the execution of theescalations. Thus, the behavior analysis module 410 comprises areal-time analysis component 412 and a historical analysis component414. Various real-time and historical behavioral analyses may beperformed by the behavior analysis module 410.

A real-time behavioral analysis that the real-time analysis component412 of the behavior analysis module 410 may perform is to interrogatethe monitored escalation list data structure 416 to determine whichescalations have a relatively large number of database transactionsevaluated by the escalation, relatively large response times, or thelike, where “relatively large” may be determined based on a comparisonof a number of database transactions evaluated by the escalation to oneor more predetermined thresholds defining the border between what isconsidered a relatively small or normal number of database transactionsand a relatively large number of database transactions. Such a thresholdmay be implementation dependent and may be set by a system administratoror other authorized user. This threshold information may be correlatedwith the configured action of the escalation point of the escalation andthe target database, database fields, or the like. In this way, theescalations that are most database intensive may be identified byreal-time analysis component 412 as candidates for automaticallysplitting the escalation so as to reduce the load on the escalationapplication and, the target database.

For example, if an escalation dealing with the data/records associatedwith users A, B, and C always takes a time to complete that is beyond apredetermined threshold, and real-time analysis component 412 determinesthat user A's corresponding escalation action shows a number of databasetransactions in the escalation log data structure 499 of the escalationapplication 498 that is beyond a predetermined threshold, then executionconditions applying to user A can be automatically removed, such as byescalation update module 460 for example, into a separate escalationwith appropriate scheduled run times. That is, rather than theescalation operating on all of the user's A, B, and C databasetransactions/database records collectively, a separate escalation may beautomatically generated, such as by escalation update module 460 forexample, that mirrors the original escalation but operates only on userA's database transactions/database records as the target. In addition,the target of the original escalation may be modified, such as byescalation update module 460 for example, to only operate on thedatabase transactions/database records of users B and C.

The real-time behavioral analysis component 412 may detect suchconditions (e.g., that an escalation dealing with the data/recordsassociated with users A, B, and C always takes a time to complete thatis beyond a predetermined threshold and that user A's correspondingescalation action shows a number of database transactions in theescalation log data structure 499 of the escalation application 498 thatis beyond a predetermined threshold) and flag them by adding anappropriate recommended action to the recommended action list datastructure 432 maintained by the recommendation module 430 to performsuch an escalation splitting action. The recommended action may actuallycomprise multiple recommended actions, e.g., one recommended action tomodify the target of the escalation and another recommended action tocreate a new escalation corresponding to the original escalation butwith the target being different from the original escalation.

The historical analysis component 414 may perform various historicalbehavioral analyses. For example, the historical analysis component 414may analyze a historical trend to create a baseline of escalationcompletion times over a period of time and determine if a delay valueassociated with an escalation is appropriate for optimum performance ofthe escalation. Whether a delay value associated with an escalation isappropriate for optimum performance of the escalation may be determinedby historical analysis component 414 by comparison against one or morepredetermined thresholds. Based on this historical trend analysis, thedelay value associated with an escalation may be updated by historicalanalysis component 414 upon detecting spikes or the like.

For example, consider an escalation (Esc123) that has a lag that variousthrough the day as follows:

-   -   at 6:00 am: 5 minute run time, finishes in 2 minutes    -   at 1:00 pm: 5 minute run time, finishes in 10 minutes    -   at 5:00 pm: 5 minute run time, finishes in 6 minutes        From this escalation execution metric information, which again        may be collected by the escalation metric measurement module 420        and provided as input to the behavior analysis module 410,        historical analysis component 414 may calculate an escalation        baseline and validate the escalation baseline against a        configurable threshold, e.g., 80% meaning that the baseline is        80% of the measured escalation execution time. For example, if        Esc123 has an average runtime for a given, day of 6 min;        (6+10+2)/3, an 80% threshold would be approximately 7.2 minutes        (baseline+20%). The configurable threshold may be configured        based on various criteria including current loading of the        system or the like.

Based on the validation against the threshold, an appropriaterecommended action to adjust one or more of the run time and/or thedelay value may be determined by historical analysis component 414 inorder to optimize the performance of the escalation. For example, if theexecution time of Esc123 in the above example is above 7.2 minutes, anappropriate recommended action may be generated by historical analysiscomponent 414. For example, an appropriate recommended action may be tostagger or reschedule the execution of the escalation to improve overallexecution time of the escalation. This may involve rescheduling theescalation's execution to a time during off-peak, i.e. lighter load, ofthe system, adjusting a delay or runtime interval, or optimizing thecondition of the escalation to improve overall run time. The recommendedaction may be added to the recommended action list data structure 432maintained by the recommendation module 430.

As another example of a history analysis that may be performed by thehistorical analysis component 414, the historical analysis component 414may detect conditions indicative of a need to monitor particularescalations that may not already be listed in the monitored escalationlist data structure 416. If such a condition is determined to bepresent, the corresponding escalation may be automatically added, by thehistorical analysis component 414, to the monitored escalation list datastructure 416. In one illustrative embodiment, the condition indicativeof a need to add a particular escalation to the monitored escalationlist data structure 416 is a condition in which an escalation failsconsistently or periodically with a predetermined threshold amount offrequency, where failure of an escalation means that the escalation ishung, does not complete its execution on all records/data it is intendedto processor, does not execute at all, or otherwise did not produceexpected results.

The escalation log information in the escalation log data structure 499,for escalations that are defined and managed by the escalationapplication 498 and which are not already listed in the monitoredescalation list data structure 416, may be periodically analyzed by thehistorical analysis component 414 to determine if the escalation loginformation for one or more of the escalations meets the conditioncriteria indicative of a need to add the escalation to the monitoredescalation list data structure 416. For example, if the escalation loginformation for a particular escalation indicates that the escalationfails consistently or periodically with at least a predeterminedthreshold frequency, the escalation may be identified by the historicalanalysis component 414 as one that needs to be monitored more closelyand should be added to the monitored escalation list data structure 416.

A determination as to whether the escalation log information for aparticular escalation indicates that the escalation fails consistentlyor periodically with at least a predetermined threshold frequency; maybe made, for example, by the historical analysis component 414 queryingescalation log information parameters such as “schedule” “last runtime”, and “status” at periodic intervals with minimal performanceimpact. A count of the number of times the “status” of the escalation isindicative of a failed escalation may be maintained by the historicalanalysis component 414 for each of the escalations for which there isescalation log information. The count may be maintained for apredetermined period of time at the end of which the count may bereinitialized to a starting value. If the count exceeds a predeterminedthreshold value, then the escalation is identified by the historicalanalysis component 414 as one that needs to be more closely monitored byadding the escalation to the monitored escalation list data structure414. An appropriate action for adding the escalation to the monitoredescalation list data structure 416 may be added, by the historicalanalysis component 414 for example, to the recommended action list datastructure 432 maintained by the recommendation module 430 in such acase.

Yet another type of historical analysis that may be performed by thehistorical analysis component 414 may be to determine whether dividingor splitting an escalation would improve performance beyond a configuredthreshold and then generate an appropriate recommended action to performsuch division or splitting. This historical analysis may be based on acalculated processing time and amount of data/number of recordsprocessed by an escalation. Essentially, in performing this historicalanalysis, historical analysis component 414 looks at the variousoptional schedules for an escalation, determines the amount ofdata/number of records processed under each optional schedule and theamount of processing time needed to execute the escalation under theseoptional schedules, and determines whether a run time interval should bemodified in the current schedule of the escalation. That is, the currentescalation may have been defined in accordance with conditions that havechanged since the current escalation's instantiation which result in theperformance of the escalation becoming less efficient. For example, theload of the escalation, e.g., number of records processed, capacity ofthe target of the escalation, and the like, may have changed causing theprocessing time to saturate and peak out after processing a smalleramount of data/number of records. Thus, if the escalation is stilloperating on a larger amount of data/number of records between scheduledrun time intervals, the execution of the escalation becomes lessefficient. The historical analysis component 414 may identify suchsituations from the historical analysis and generate recommended actionsto improve the performance of the escalation by dividing or splittingthe escalation such that more instances of the escalation are executedon a more frequent basis on a smaller amount of data/number of records.

For example, assume that an escalation is set to execute with a run timeinterval of every 2 hours, and the execution of this escalation resultsin the following historical trend over a 1 week period:

Number of records processed: average of 10,000 records

Processing time: average of 90 minutes

Assume also that a similar escalation, as may be determined by variouscharacteristics of the escalation including the escalation operating ona same number of records/data, targeting a same set of database objects,executing for a same customer/implementer of the escalation, or the like(similarity may be determined by applying escalations against the targetdatabase objects/data set, generating counts and analyzing conditions ofthe escalations to determine which escalations are similar), at a runtime interval of every 30 minutes, has a historical trend that yields abetter, processing time as follows:

Number of records processed: average of 5,000 records

Processing time: average of 20 minutes

In other words, with the current escalation being analyzed, 10,000records are processed in 90 minutes of processing time while in thesimilar escalation, 22,500 records may be processed in 90 minutes ofprocessing time by reducing the run time interval to every 30 minutesand operating on relatively smaller amounts of data/numbers of recordsduring each execution. From this, the historical analysis component 414may determine that the current escalation under analysis may be dividedor split into additional instances of the escalation in a schedule ofthe execution of the escalation. That is, since the similar escalationobtains a shorter processing time when run at a shorter run timeinterval, then a similar trend may be expected with the currentescalation if it is divided and split into additional escalationinstances that are executed every 30 minutes as opposed to every 2hours. As a result of such a determination, the historical analysiscomponent 414 may generate an appropriate recommended action to modifythe run time interval of an escalation. For example, a recommendedaction may be to reduce the run time interval to a smaller interval andto adjust the escalation's scheduled to reflect the new run timeinterval.

It should be appreciated that while the above embodiments are describedin terms of splitting an escalation based on the number of recordsprocessed and execution time of a similar escalation, the illustrativeembodiments are not limited to such. Rather, splitting of escalationsmay be performed based on timestamps, status, ownergroup, or any otherattribute included in the escalation condition.

Thus, the behavior analysis module 410 analyzes the behavior ofescalations as identified by the escalation execution metric informationthat is generated by the escalation application 498 by monitoringescalation executions and that is collected by the escalation metricmeasurement module 420. The escalation metric measurement module 420gathers such metric information from the escalation log data structure499 and may also perform some pre-processing of this metric measurementdata to provide data useful for the analysis of behavior by the behavioranalysis module 410. The escalation metric measurement module 420 mayalso gather load, system, and application data for systems andapplications across application and database servers, including replicadatabase servers. System data may include CPU, disk I/O, process, andother data indicative of the operation of the hardware systems.Application data may include concurrent user, heap, paging, and NMmetrics, for example. This information together may be indicative of aload on the systems and may be correlated, such as by the escalationmetric measurement module 420 for example, with escalation executiontime in order for historical analysis component 414 to makerecommendations as to how to improve the execution of an escalation,e.g., if an escalation execution is slow because a system is heavilyloaded, rescheduling of the escalation may be recommended as opposed tosplitting the escalation.

The behavior analysis module 410 performs one or more real-timebehavioral analysis of the escalation metric measurements and/or one ormore historical behavioral analysis of the escalation metricmeasurements, and generates one or more recommended actions that areinserted or added, by the behavior analysis module 410, to a recommendedaction list data structure 432 maintained by the recommendation module430. The recommended actions may comprise actions for adjusting delayvalues, run time intervals, targets, and other parameters ofescalations, splitting an escalation, generating new escalations, andthe like.

Recommended actions in the recommended actions list data structure 432of the recommendation-module 430 are submitted to anapproval/confirmation process via the approval/confirmation module 440.That is, the approval/confirmation module 440 may present therecommended actions to a system administrator or other authorized uservia one or more user interfaces such that the authorized user is able toapprove/confirm/deny the recommended action. The'one or more userinterfaces may be presented to the authorized user in any one or more ofa plurality of different ways including a web page accessed by theauthorized user, an application present on a computing device used togenerate an output based on notifications sent by the recommendationmodule 430, a electronic mail communication transmitted to theauthorized user's computing device, a SMS communication transmitted tothe authorized user's computing device, or the like.

The authorized user may respond to such notifications of recommendedactions via the user interface to authorize/confirm or deny execution ofthe recommended action. If a recommended action is authorized/confirmed,then a corresponding indication of this authorization may be added, bythe approval/confirmation module 440, to the recommended action listdata structure 432. Similarly, if the recommended action is a denial ofthe recommended action, a corresponding indication of this denial may beadded, by the approval/confirmation module 440, to the recommendedaction in the recommended action list data structure 432.

In one illustrative embodiment, the presentation of the recommendedactions to the user for approve/confirm/deny the recommended action mayfurther provide the user with options to apply the recommended actionimmediately, apply the recommended action at a next execution of theescalation, apply the action at a later scheduled time, or the like. Forexample, if a user selects to apply the action at a later scheduledtime, the later scheduled time may be a time of relatively lower load onthe system or a predetermined change window time. An indication of theuser's selection to approve/confirm/deny the recommended actions,whether to apply the recommended actions immediately, at a nextexecution, or at a later scheduled time, and the like, may be stored byapproval/confirmation module 440 in association with an identifier ofthe recommended action for later analysis.

In addition, the approval/confirmation module 440 may store a list ofpre-approved recommended actions that may be used to automaticallyapprove/confirm particular recommended action types. Thus, when arecommended action is inserted or added to the recommended action listdata structure 432, the type of the recommended action, e.g., a nm timeinterval modification, delay value modification, escalation split, orthe like, may be compared to the pre-approved recommended action types.If there is a match between the type of the recommended action added tothe action list and one of the pre-approved recommended action types,then the recommended action that is added to the action list may beautomatically authorized/confirmed and a corresponding indicator may beadded, by the approval/confirmation module 440, to the recommendedaction list data structure 432 in association with the recommendedaction.

The simulation module 450 periodically accesses the recommended actionslist data structure 432 of the recommendation module 430, and retrievesthe recommended actions that have been authorized/confirmed as indicatedby the authorized/confirmed indication in association with therecommended actions in the recommended actions list of therecommendation module 440. The simulation module 450 may also removethose recommended actions in the recommended actions list data structurethat have been denied, as indicated by an associated denial indicator,or that have become stale as simulation module 450 may determine from atimestamp associated with the recommended action and one or morethresholds.

For the recommended actions that have been authorized/confirmed, thesimulation module 450 may simulate the recommended action, in a testenvironment, to determine whether the recommended action should becommitted to the escalation application. That is, the simulation module450 monitors the execution of a modified or new escalation, generated,by the escalation update module 460, as a result of the application of arecommended action, in a test environment, and the simulation module 450determines whether the execution of the modified or new escalationconstitutes an improvement in the execution of the escalation over theoriginal, unmodified escalation. This simulation may be performed on asimulated system; virtual machine, development instance, or the like, ofthe database or other system with which the escalation application 498operates. Infrastructure, security, and network access is in place tosupport such simulations and to obtain results of such simulations suchthat the results may be presented to simulation module 450 for automatedanalysis and/or an authorized user for review prior to committing therecommended actions to the escalations of the escalation application498.

For example, simulation module 450, in performing the automatedanalysis, may obtain results of the simulation of the modifiedescalation to determine the measured performance of the modifiedescalation. The measured performance of the modified escalation asobtained from this simulation may be compared to the actual baselineperformance measurements of the current escalation (described above) todetermine if a significant enough improvement in performance is achievedby implementing the recommended action, where significance of theimprovement may be measured according to one or more threshold values,e.g., a 20% improvement (reduction) in processing time required toexecute the escalation, a 25% improvement (increase) in amount of dataor number of records processed by the escalation within a given periodof time, or any of a plethora of other possible threshold values orcombinations of threshold values. The result is a set of one or moredecisions as to whether to commit the recommended action to theidentified escalation in the escalation application 498.

Such evaluations may be done using an automated tool. In other cases,such determinations may be made by a system administrator or otherauthorized user in which ease the results of the simulation may beoutput to the system administrator, optionally in combination withbaseline performance measurements, and the systemadministrator/authorized user may indicate via a user interface whetherthe recommended action should be committed to the escalation in theescalation application 498.

Based on the determination as to whether to commit the recommendedaction or not, a commit identifier may be added, by the control logic405 or other appropriate logic in the proactive monitoring and dynamicoptimization engine 400, to the entry in the recommended action listdata structure 432. This identifier is a flag to the escalation updatemodule 460 to either perform the update on the escalation or not when ascheduled update of escalations occurs. That is, the escalation updatemodule 460 may periodically interrogate the recommended action list datastructure 432 for recommended actions that have their commit identifierset to a value indicative of a need to commit the action to theescalation in the escalation application. For those recommended actionsin the recommended action list data structure 432 having a set commitidentifier, the escalation update module 460 performs operations inconjunction with the escalation application 498 to modify, split,create, remove, or the like, the corresponding escalation in theescalation application. In order to perform such operations, theescalation update module 460 takes into consideration the productionenvironment which may require change tickets to be generated, request achange window of time in which to perform the changes, etc.

In some instances, the escalation update module 460 may dynamicallyswitch the virtual machines, processors, or other resources on which theescalation runs in order to perform load balancing as well. That is,relative loads on the various virtual machines run by the systemassociated with the escalation application 498 may be monitored and if avirtual machine is determined to be relatively less loaded than othervirtual machines, escalations whose modifications are indicative of alower performance due to processor time or the like, may be migrated byescalation update module 460 to the virtual machines having relativelylower loads.

In addition to the detection of the possible need for modifications toescalations based on real-time and historical trend analysis of theexecution performance information of the various escalations, theillustrative embodiments further provide another real-time analysis fordetecting “hung” escalations, as previously defined. The illustrativeembodiments provide logic for detecting such hung escalations andcorrecting them to avoid the hung condition. The embodiments fordetecting and correcting hung escalations may operate separate from theother embodiments described above or may operate in conjunction withthese other embodiments in either a parallel or serial manner. Thus, insome illustrative embodiments, the detection and correction of hungescalations may be performed at substantially a same time as the otherbehavioral analysis and update of escalations previously described.Other illustrative embodiments for detecting and correcting hungescalations may operate on results of the behavior analysis generated byembodiments described above, the simulation results generated by thesimulation module 450, or the like, in a more serial manner.

The hung-escalation detection logic may be embodied in a hung escalationdetector 470, for example, which is responsible for detectingescalations that either do not complete (fail) or complete beyond one ormore predetermined allotted processing time threshold. The hungescalation detector 470 may store one or more predetermined allottedtime processing thresholds which are used as a basis for comparisonagainst measured processing time metric information gathered by theescalation metric measurement module 420 from the escalation log datastructure 499 of the escalation application 498. The one or morepredetermined allotted processing time thresholds may be associated withdifferent types of escalations such that one escalation in theescalation application 498 may have a different correspondingpredetermined allotted processing time threshold than another escalationin the escalation application 498. These predetermined processing timethresholds may be automatically generated, such as by the behavioranalysis module 410 for example, based on baseline calculationsperformed by the behavior analysis module 412 as described above, or maybe user defined and set in the hung escalation detector 470.

The hung escalation detector 470 may analyze the measured metricinformation gathered by the escalation metric measurement module 420 toidentify any escalations that did not complete correctly, as may bedetermined from the escalation log data structure 499, a“post-verification” check of the execution of the escalation, or thelike. Those escalations identified as not having completed correctlyhave an associated recommended action added to the recommended actionlist data structure 432 of the recommendation module 430 (if one is notalready present), with a corresponding “hung” identifier set in therecommended action entry or an already existing recommended action forthis escalation is further updated to have the hung identifier setindicating that the escalation is hung and needs to be corrected.

In addition, the hung escalation detector 470 detects escalations thatcomplete, but complete beyond an allotted processing time threshold. Forexample, if an escalation is scheduled to execute at N time intervals(e.g., every 2 hours, every 30 minutes, etc.), then the hung escalationdetector 470 may determine if the sum of the last run time (asdetermined from the escalation metric measurements obtained from theescalation log data structure), plus the N interval, plus a delay valueis greater than a corresponding allotted processing time threshold forthe type of escalation. If so, then hung escalation detector 470determines that the escalation is hung. If not, then the escalation isnot hung. If the escalation is determined to be hung because itcompletes in a time that is greater than the allotted processing timethreshold for that type of escalation, then hung escalation detector 470may generate and/or update a recommended action such that the “hung”identifier is set in the recommended action entry.

For example, assume that the current time is 19:00 UTC and an escalationis scheduled to run every 6 hours, starting at 6:00 UTC (i.e. 6, 12, 18,and 24). The last run time is determined to be 12:00 UTC and the delayallowed (delay value) is 30 minutes. The last run time should be between18:00 and 18:30 UTC. If the actual last run time of the escalation isdetermined to be earlier than 18:00, then it may be determined that theescalation failed to complete and thus, is considered hung. If the lastrun time+the interval (6 hours)+the allowed delay (delay value of 30minutes) is greater than a threshold, e.g., 19:00 UTC, then theescalation may be determined to be hung due to it taking a longer thanallotted processing time to complete.

In addition, the hung escalation detector 470 may further detectsituations where an escalation completes within the allotted processingtime threshold for the type of escalation, but the defined action in theescalation is not performed, e.g., data is not updated, a record in adatabase is not updated, a notification is not sent, etc. The hungescalation detector 470 comprises logic for cross-checking the resultsof the actions defined in the escalations against the escalationexecution log data structure 499 to detect such situations and identifythem as hung escalations, i.e. perform a “post-verification” checkoperation that verifies that the escalation completed correctly. As withthe embodiments described above, in such a situation an appropriaterecommended action may be added by hung escalation detector 470 to therecommended action list data structure 432 and/or the recommended actionmay be updated to have its hung identifier set.

The escalation fix module 490 periodically interrogates the recommendedaction list data structure 432 for recommended actions having the hungidentifier set. For those escalations corresponding to recommendedactions having a hung identifier set, the escalation fix module 490 mayperform a process for fixing the escalation to avoid the hung condition.Such fixes may include splitting the escalation, modifying parameters ofthe escalation, and/or the like. Filters may be applied to therecommended actions that have their hung identifiers set such that notall of these recommended actions are necessarily fixed by the escalationfix module 490. For example, a filter of a run interval of less than 30minutes may be used as a basis for filtering the recommended actionswith the hung identifier set. That is, only recommended actions with ahung identifier set and that are associated with escalations having arun interval of 30 minutes or less are fixed by the escalation fixmodule 490 in this example. Any filter criteria or no filter criteriamay be used without departing from the spirit and scope of theillustrative embodiments.

In one illustrative embodiment, the escalation fix module 490 mayoperate by duplicating the existing escalation, renaming the newescalation using a standardized naming convention, updating escalationconditions of this new escalation to work only on the targetdata/records that were not yet processed by the original escalation,disabling the hung escalation, and enabling/activating the newescalation.

In one illustrative embodiment, as with the other recommended actionsnoted above, the modifications to be performed by the escalation fixmodule 490 may be submitted as recommended actions which are thenapproved/confirmed/denied via the approval/confirmation module 440.Thus, an automated process or authorized user may need to authorize themodifications or “fixes” performed by the escalation fix module 490prior to the modifications or fixes being committed. The simulationmodule 450 may likewise be used to simulate the modifications or fixesprior to the modifications or fixes being committed. Alternatively, theescalation fix module 490, due to the nature of the hung condition beingdifferent from the more “optional” modifications of the other behavioralbased recommended actions, may operate outside of theapproval/confirmation module 440, simulation module 450, and escalationupdate module 460 and may operate automatically in response to thedetection of hung escalations.

It should be noted that when a recommended action is completed, thecorresponding recommended action may be removed by a cleanup operationfrom the recommended action list data structure 432. In addition, acleanup operation may be run periodically or at scheduled times toremove recommended actions that have become stale, i.e. have not beencommitted within a predetermined period of time. In this way, therecommended action list data structure 432 may be maintained.

The escalation staggering module 495 operates to minimize hungescalation situations and to improve performance of escalations bystaggering the run times of the escalations by modifying the schedulesof the escalations. That is, escalations having a similar schedule, andrun, interval are identified and their schedules and run intervals areupdated or modified so as to stagger them. The similar schedules and runintervals may be determined in accordance with a configuration file (notshown) or the like, associated with the proactive monitoring and dynamicoptimization engine 400, that stores information about the configurationof the escalations including their scheduled execution times and timeintervals.

In one illustrative embodiment, the escalation staggering module 495 mayoperate to determine, from the configuration file, a number ofescalations scheduled for simultaneous execution in a time interval in aproduction environment. The escalation staggering module 495 may dividethe time interval by the number of escalations to form a shorted timeinterval and then may reschedule execution of the number of escalationsin the production environment such that a plurality of subsets of thenumber of escalations execute in a staggered order according to theshortened time interval. A subset may be a subset of the escalationsthat all share a common characteristic, such as a frequency at which theescalations in the subset failed to complete execution or completedexecution beyond an allotted processing time, a load, a customer, anotification, a service level agreement, and a database object.

Furthermore, before rescheduling the execution of the number ofescalations in the production environment, the escalation staggeringmodule 495 may reschedule the execution of the number of escalations ina test environment. This may be done in response to a determination thatperformance of the rescheduled execution of the number of escalations inthe test environment is improved over performance of simultaneousexecution of the number of escalations in the production environment.

For example, assume that the starting time is the same for escalations1, 2, 3, 4, and 5 and the each of these escalations is set to run every10 minutes. Without staggering the schedules of these escalations, eachof these escalations would be scheduled to execute at the same time orsubstantially a same time and run for substantial a same amount of time.This may cause the performance of these escalations to be less thanoptimal.

The escalation staggering module 495 staggers the schedules of theseescalations such that the start times of these escalations are notidentical. For example, escalation 1 may have a start time at minute 0,escalation 2 may have a start time at minute 2, and escalation 3 mayhave a start time at minute 4, etc. Each escalation will still run every10 minutes, but with different starting times. Thus, the staggeredschedule provides less load on the system and allows for improvedperformance of the escalations. The affects of such staggering areexponential with large numbers of escalations having same or similar runtime intervals and start times. Moreover, the escalation staggeringmodule 495 may classify escalations according to escalation types andthen stagger the escalations according to classification. For example,escalations may be classified into predetermined classifications basedon such characteristics as hung frequency, load, customer, notification,SLA, target, or the like. With such groupings, for example, notificationescalations may be scheduled for minute 0, SLA escalations may bescheduled for minute 2, object “WORKORDER” escalations may be scheduledfor minute 4, and the like.

The alert/notification module 480 automatically generates notificationsand transmits them to computing devices associated with authorized usersso as to inform them of conditions, actions, and the like, occurring inthe proactive monitoring and dynamic optimization engine 400. Suchnotifications may take many different forms including, but not limitedto, electronic mail notifications, paging notifications, SMSnotifications, automated telephone calls, instant messagingnotifications, displaying user interfaces, and the like.

Thus, the illustrative embodiments provide mechanisms for automaticallyimproving the performance of escalations based on behavioral analysis ofthe escalations. This behavioral analysis comprises real-time behavioralanalysis, historical trend behavioral analysis, and hung escalationanalysis. As noted above, while the illustrative embodiments aredescribed with regard to escalations, the illustrative embodiments maybe applied to any computing system in which the execution behavior ofscheduled tasks may be analyzed using the illustrative embodiments andappropriate recommended actions for improving the performance of thesescheduled tasks may be generated and committed.

As discussed above, one of the principle data structures utilized by theillustrative embodiments is the recommended action list data structure432 of the recommendation module 430. FIG. 5 is an example diagram ofone recommended action list entry that may be used with this recommendedaction list data structure in accordance with one illustrativeembodiment. As shown in FIG. 5, the recommended action list entry 500includes a field 510 specifying an identifier of an escalation withwhich the recommended action is associated. A second field 520 isprovided for specifying the type of recommended action, e.g., splittingof the escalation, modifying a run time interval of the escalation,modifying a delay value of the escalation, modifying a target of theescalation, etc. A third field 530 is provided for specifying adescription of the recommended action and the basis for the recommendedaction. A fourth field 540 is provided for specifying the parametervalues, if any, of the escalation and their corresponding updated valuesshould the recommended action be committed. A fifth field 550 isprovided for specifying an approval/denial of the recommended action,this field being set via the approval/confirmation module 440 asdescribed above. A sixth field 560 is provided for specifying whether ornot to commit the recommended action. This field may be set bysimulation module 450 as described above. A seventh field 570 isprovided for specifying whether the recommended action is associatedwith a hung escalation. This field may be set by the hung escalationdetector 470 as described above.

FIG. 6 is a flowchart outlining an example operation for performing areal-time behavior analysis of an escalation in accordance with oneillustrative embodiment. The operation outlined in FIG. 6 may beperformed, for example by the real-time analysis component 412 of thebehavior analysis module 410 of the proactive monitoring and dynamicoptimization engine 400, for example. The operation outlined in FIG. 6is for a single escalation and may be repeated for each escalation in amonitored escalation list.

As shown in FIG. 6, the operation starts by real-time analysis component412 determining a number of transactions handled by the escalation sincea last time that the real-time behavioral analysis was performed orwithin a designated time interval (step 610). Real-time analysiscomponent 412 determines an action and field association between thenumber of transactions and the measured metrics of the escalation (step620) and real-time analysis component 412 correlates this data (step630). Real-time analysis component 412 makes a determination based onthis data as to whether the escalation execution is resource intensive(step 640). If not, the operation terminates. If the escalationexecution is resource intensive, then real-time analysis component 412identifies the escalation for splitting (step 650) and real-timeanalysis component 412 adds a corresponding recommended action to therecommended action list data structure 432 (step 660). The operationthen terminates.

FIG. 7 is a flowchart outlining an example operation for performing ahistorical trend behavior analysis of an escalation in accordance withone illustrative embodiment. The operation outlined in FIG. 7 may beperformed, for example by the historical analysis component 414 of thebehavior analysis module 410 of the proactive monitoring and dynamicoptimization engine 400, for example. The operation outlined in FIG. 7is for a single escalation and may be repeated for each escalation in amonitored escalation list.

As shown in FIG. 7, the operation starts by historical analysiscomponent 414 analyzing the escalation log information (e.g., inescalation log data structure 499) to identify a historical trend in theexecution of the escalation (step 710). Historical analysis component414 calculates a baseline for the performance of the escalation (step720) and correlates the baseline with measured metrics for the executionof the escalation (step 730). Historical analysis component 414 makes adetermination as to whether a difference between the baseline and themeasured metrics for the execution of the escalation meets or exceedsone or more predetermined thresholds (step 740). If not, the operationterminates. If so, then historical analysis component 414 identifies theescalation for an update of its execution parameters, e.g., delay value,run interval, target, schedule, etc (step 750). Historical analysiscomponent 414 then adds one or more corresponding recommended actions tothe recommended action list data structure 432 (step 760) and theoperation terminates.

FIG. 8 is a flowchart outlining an example operation for performing hungescalation analysis in accordance with one illustrative embodiment.Aspects of the operation outlined in FIG. 8 may be performed, forexample by the hung escalation detector 470, escalation fix module 490,and escalation staggering module 495 of the proactive monitoring anddynamic optimization engine 400, for example. The operation outlined inFIG. 8 is for a single escalation and may be repeated for eachescalation in a monitored escalation list.

As shown in FIG. 8, the operation starts by hung escalation detector 470retrieving measure metric information for the escalation, such as fromthe escalation metric measurement module 420, escalation log datastructure 499, or the like (step 810). Hung escalation detector 470analyzes the measured metric information to determine if any hungconditions are met (step 820). Hung escalation detector 470 makes adetermination is made as to whether the escalation is hung (step 830).If not, the operation terminates. If so, then hung escalation detector470 generates a corresponding recommended action to fix the hungcondition and/or updates an existing recommended action for thisescalation to indicate the hung condition and a need to fix theescalation for the hung condition (step 840).

The process may then go to the approval process described above andsummarized in FIG. 9 hereafter. Escalation fix module 490 makes adetermination as to whether the fix of the hung condition for thisescalation has been approved (step 850). If not, the operationterminates. If so, then escalation fix module 490 fixes the hungescalation, e.g., escalation fix module 490 executes the recommendedaction (step 860), and escalation staggering module 495 staggersperformance of the updated escalation if determined to be necessary(step 870). The operation then terminates.

FIG. 9 is a flowchart outlining an example'operation for performing anapproval/commit operation in accordance with one illustrativeembodiment. Aspects of the operation outlined in FIG. 9 may beperformed, for example, by approval/confirmation module 440, simulationmodule 450, the escalation update module 460, and escalation staggeringmodule 495 of the proactive monitoring and dynamic optimization engine400.

As shown in FIG. 9, the operation starts by approval/confirmation module440 retrieving the next recommended action from the recommended actionlist data structure 432 (step 910). Approval/confirmation module 440presents the recommended action for approval/denial (step 912). This mayinvolve an automated approval/denial process and/or the sending ofnotifications to an authorized user requesting approval/denial andreceiving a user input indicating approval/denial.

Approval/confirmation module 440 makes a determination as to whether therecommended action is approved or not (step 914). If the recommendedaction is approved, simulation module 450 executes a simulation of theupdated escalation executed (step 916) and a determination is made as towhether the results of the simulation indicate a significant enoughimprovement in the performance of the escalation to warrant committingthe update to the escalation (step 918). If so, then escalation updatemodule 460 updates the recommended action to indicate that therecommended action should be committed (step 920).

A determination is then made as to whether there are more recommendedactions to evaluate (step 922). If so, then the operation returns tostep 910. If there are no more recommended actions to evaluate, thenescalation update module 460 searches the recommended actions toidentify recommended actions that have the commit identifiers set (step924). For each of these recommended actions, escalation update module460 commits the actions on the corresponding escalations in theescalation application (step 926). Escalation staggering module 495performs staggering of the schedule of these escalations is ifdetermined to be necessary (step 928). The operation then terminates.

Thus, the illustrative embodiments provide for improving performance ofan escalation. With these embodiments, a computer monitors performanceof an escalation in a production environment. The computer identifies acharacteristic of the escalation based on the monitored performance andcreates a recommendation for improving escalation performance based onthe characteristic. In response to an approval of the recommendation,the computer applies the recommendation to the escalation to form one ormore recommended escalations. The computer deploys the one or morerecommended escalations into the production environment.

In one illustrative embodiment, before the computer deploys the one ormore recommended escalations into the production environment, thecomputer monitors performance of the one or more recommended escalationsin a test environment. The computer deploys the one or more recommendedescalations into the production environment in response to adetermination that the monitored performance of the one or morerecommended escalations in the test environment is an improvement overthe monitored performance of the escalation in the productionenvironment.

In one illustrative embodiment, the computer identifies thecharacteristic of the escalation based on the monitored performance byreading a log data structure associated with the escalation anddetermining from the log data structure that the escalation isassociated with first and second customers. The computer identifies inthe log data structure a number of database transactions performed bythe escalation for the first customer. The characteristic is that thenumber of database transactions performed by the escalation for thefirst customer exceeds a threshold. The recommendation comprisesdisassociating the first customer from the escalation, splitting theescalation into first and second escalations, and associating the secondescalation with the first customer.

In another illustrative embodiment, the characteristic is that one ormore durations of execution of the escalation vary from an expectedduration of execution of the escalation by a threshold and therecommendation comprises adjusting an execution time of the escalation.In still another illustrative embodiment, the characteristic is that theescalation historically processes records in a database at a rate thatis slower than a rate at which another escalation processes records inthe database and the recommendation comprises executing the escalationat a frequency that is higher than a frequency at which the escalationis configured to execute.

In one illustrative embodiment, before the computer monitors performanceof the escalation in the production environment, the computer queriesparameters associated with the escalation, determines from theparameters that the escalation generated a number of errors over aperiod of time at a rate that exceeds a threshold, and adds theescalation to a list of escalations for performance monitoring.Moreover, the computer may gather load data for use in identifying thecharacteristic.

In another illustrative embodiment, after the computer deploys the oneor more recommended escalations into the production environment, thecomputer determines that a defined action of a recommended escalation ofthe one or more recommended escalations did not execute. In response,the computer notifies a user that the defined action of the recommendedescalation did not execute.

In other illustrative embodiments, mechanisms for minimizing occurrencesof hanging escalations in a computer system are provided. With theseillustrative embodiments, a computer determines that a number ofescalations are scheduled for simultaneous execution in a time intervalin a production environment. The computer divides the time interval bythe number of escalations to form a shortened time interval. Thecomputer reschedules execution of the number of escalations in theproduction environment such that a plurality of subsets of the number ofescalations execute in a staggered order according to the shortened timeinterval. A hanging escalation is an escalation that fails to complete,fails to process all data or records that the escalation was to process,or completes beyond an allotted processing time.

The computer may receive a configuration defining a subset of theplurality of subsets, the escalations in the subset sharing a commoncharacteristic. The common characteristic may be selected from the groupconsisting of a frequency at which the escalations in the subset failedto complete execution, a frequency at which the escalations in thesubset failed to process all data or records that the escalations wereto process, a frequency at which the escalations in the subset completedexecution beyond an allotted processing time, a load, a customer, anotification, a service level agreement, and a database object.

In one illustrative embodiment, before the computer reschedules theexecution of the number of escalations in the production environment,the computer reschedules the execution of the number of escalations in atest environment. The computer may then reschedule the execution of thenumber of escalations in the production environment in response to adetermination that performance of the rescheduled execution of thenumber of escalations in the test environment is improved overperformance of simultaneous execution of the number of escalations inthe production environment. In one illustrative embodiment, the computerreschedules execution of the number of escalations in the productionenvironment such that each escalation in the number of escalations has adifferent execution start time. Each of the escalations in the number ofescalations may have a same execute time period.

In one illustrative embodiment, the computer classifies escalations ofthe number of escalations into a plurality of escalation types. Each ofthe subsets in the plurality of subsets may be associated with at leastone escalation type in the plurality of escalation types. The computermay classify escalations of the number of escalations into a pluralityof escalation types based on characteristics of the escalations. Thecharacteristics of the escalations may comprise at least one of a hungfrequency, a load of the escalation, a customer associated with theescalation, a notification generated by the escalation, a service levelagreement (SLA) associated with the escalation, or a target of theescalation. Escalations of a same escalation type may be scheduled tostart at a same start time.

In another illustrative embodiment, mechanisms for processing work itemsthat have not been completed by a first escalation are provided. Withthis illustrative embodiment, a computer determines that the firstescalation, failed to complete execution, processed fewer work itemsthan the first escalation is configured to process, or completedexecution beyond an allotted processing time. The computer duplicatesthe first escalation to form a second escalation. The computerconfigures the second escalation to process the work items that have notbeen completed by the first escalation. The computer disables the firstescalation and activates the second escalation to process the work itemsthat have not been completed by the first escalation.

The detection of the first escalation having failed to completeexecution, failed to process the work items, or completed executionbeyond an allotted processing time may comprise the computer determiningthat a current time is greater than a sum of a last time that the firstescalation executed, a time interval between consecutive scheduledexecutions of the first escalation, and an allowed time period forcompletion of execution of the first escalation. The allowed time periodfor completion of execution of the first escalation is a predeterminedtime period value. The allowed time period for completion of executionof the first escalation may be calculated based on previous results ofone or more previous executions of the first escalation.

The determining operation may comprise the computer determining that thefirst escalation processed fewer work items than the first escalation isconfigured to process. The determining operation may comprise thecomputer determining that the first escalation failed to complete bycomparing a processing time metric for the first escalation in a logdata structure to a stored threshold value stored in the computer. Thedetermining operation may comprise the computer detecting that the firstescalation failed to complete by determining that the first escalationcompleted within an allotted processing time but failed to perform adefined action in the first escalation.

The computer may store a plurality of threshold values. Each thresholdvalue may be associated with a different type of escalation. The storedthreshold value may be a threshold value corresponding to a type ofescalation of the first escalation.

The computer may determine a recommendation for modifying the firstescalation. The computer may apply the recommendation to the secondescalation to generate a modified second escalation. The computer maygenerate a notification to a user requesting that the user approveapplication of the recommendation to the second escalation. Thenotification may request that the user indicate whether to apply therecommendation immediately, apply the recommendation at a next executionof the second escalation, or apply the recommendation at a laterscheduled time.

As noted above, it should be appreciated that the illustrativeembodiments may take the form of an entirely hardware embodiment, anentirely software embodiment or an embodiment containing both hardwareand software elements. In one example embodiment, the mechanisms of theillustrative embodiments are implemented in software or program code,which includes but is not limited to firmware, resident software,microcode, etc.

A data processing system suitable for storing and/or executing programcode will include at least one processor coupled directly or indirectlyto memory elements through a system bus. The memory elements can includelocal memory employed during actual execution of the program code, bulkstorage, and cache memories which provide temporary storage of at leastsome program code in order to reduce the number of times code must beretrieved from bulk storage during execution.

Input/output or I/O devices (including but not limited to keyboards,displays, pointing devices, etc.) can be coupled to the system eitherdirectly or through intervening I/O controllers. Network adapters mayalso be coupled to the system to enable the data processing system tobecome coupled to other data processing systems or remote printers orstorage devices through intervening private or public networks. Modems,cable modems and Ethernet cards are just a few of the currentlyavailable types of network adapters.

The description of the present invention has been presented for purposesof illustration and description, and is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the art. Theembodiment was chosen and described in order to best explain theprinciples of the invention, the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method for improving escalation performance,the method comprising the steps of: a computer monitoring performance ofescalation in a production environment; the computer identifying acharacteristic of the escalation based on the monitored performance; thecomputer creating a recommendation for improving escalation performancebased on the characteristic; in response to an approval of therecommendation, the computer applying the recommendation to theescalation to form one or more recommended escalations; and the computerdeploying the one or more recommended escalations into the productionenvironment.
 2. The method of claim 1, further comprising the steps of:before the step of the computer deploying the one or more recommendedescalations into the production environment, the computer monitoringperformance of the one or more recommended escalations in a testenvironment; and wherein the step of the computer deploying the one ormore recommended escalations into the production environment isresponsive to a determination that the monitored performance of the oneor more recommended escalations in the test environment is animprovement over the monitored performance of the escalation in theproduction environment.
 3. The method of claim 1, wherein: the step ofthe computer identifying the characteristic of the escalation based onthe monitored performance comprises the steps of: the computer reading alog data structure associated with the escalation; the computerdetermining from the log data structure that the escalation isassociated with first and second customers; and the computer identifyingin the log data structure a number of database transactions performed bythe escalation for the first customer; the characteristic is that thenumber of database transactions performed by the escalation for thefirst customer exceeds a threshold; and the recommendation comprises:disassociating the first customer from the escalation; splitting theescalation into first and second escalations; and associating the secondescalation with the first customer.
 4. The method of claim 1, wherein:the characteristic is that one or more durations of execution of theescalation vary from an expected duration of execution of the escalationby a threshold; and the recommendation comprises adjusting an executiontime of the escalation.
 5. The method of claim 1, wherein: thecharacteristic is that the escalation historically processes records ina database at a rate that is slower than a rate at which anotherescalation processes records in the database; and the recommendationcomprises executing the escalation at a frequency that is higher than afrequency at which the escalation is configured to execute.
 6. Themethod of claim 1, further comprising: before the step of the computermonitoring performance of the escalation in the production environment:the computer querying parameters associated with the escalation; thecomputer determining from the parameters that the escalation generated anumber of errors over a period of time at a rate that exceeds athreshold; and the computer adding the escalation to a list ofescalations for performance monitoring.
 7. The method of claim 1,further comprising the computer gathering load data for use inidentifying the characteristic.
 8. The method of claim 1, furthercomprising: after the step of the computer deploying the one or morerecommended escalations into the production environment, the computerdetermining that a defined action of a recommended escalation of the oneor more recommended escalations did not execute, and in response, thecomputer notifying a user that the defined action of the recommendedescalation did not execute.
 9. A system for improving escalationperformance, the system comprising: one or more processors, one or morecomputer-readable memories, and one or more computer-readable tangiblestorage devices; program instructions, stored on at least one of the oneor more computer-readable tangible storage devices for execution by atleast one of the one or more processors via at least one of the one ormore computer-readable memories, to monitor performance of an escalationin a production environment; program instructions, stored on at leastone of the one or more computer-readable tangible storage devices forexecution by at least one of the one or more processors via at least oneof the one or more computer-readable memories, to identify acharacteristic of the escalation based on the monitored performance;program instructions, stored on at least one of the one or morecomputer-readable tangible storage devices for execution by at least oneof the one or more processors via at least one of the one or morecomputer-readable memories, to create a recommendation for improvingescalation performance based on the characteristic; programinstructions, stored on at least one of the one or morecomputer-readable tangible storage devices for execution by at least oneof the one or more processors via at least one of the one or morecomputer-readable memories, to apply the recommendation to theescalation to form one or more recommended escalations in response to anapproval of the recommendation; and program instructions, stored on atleast one of the one or more computer-readable tangible storage devicesfor execution by at, least one of the one or more processors via atleast one of the one or more computer-readable memories, to deploy theone or more recommended escalations into the production environment. 10.The system of claim 9, further comprising: program instructions, storedon at least one of the one or more computer-readable tangible storagedevices for execution by at least one of the one or more processors viaat least one of the one or more computer-readable memories, to monitorperformance of the one or more recommended escalations in a testenvironment before deploying the one or more recommended escalationsinto the production environment; wherein the program instructions todeploy the one or more recommended escalations into the productionenvironment deploy the one or more recommended escalations into theproduction environment in response to a determination that the monitoredperformance of the one or more recommended escalations in the testenvironment is an improvement over the monitored performance of theescalation in the production environment.
 11. The system of claim 9,wherein: the program instructions to identify the characteristic of theescalation based on the monitored performance: read a log data structureassociated with the escalation; determine from the log data structurethat the escalation is associated with first and second customers; andidentify in the log data structure a number of database transactionsperformed by the escalation for the first customer; the characteristicis that the number of database transactions performed by the escalationfor the first customer exceeds a threshold, and the recommendationcomprises: disassociating the first customer from the escalation;splitting the escalation into first and second escalations; andassociating the second escalation with the first customer.
 12. Thesystem of claim 9, wherein: the characteristic is that one or moredurations of execution of the escalation vary from an expected durationof execution of the escalation by a threshold, and the recommendationcomprises adjusting an execution time of the escalation.
 13. The systemof claim 9, wherein: the characteristic is that the escalationhistorically processes records in a database at a rate that is slowerthan a rate at which another escalation processes records in thedatabase, and the recommendation comprises executing the escalation at afrequency that is higher than a frequency at which the escalation isconfigured to execute.
 14. The system of claim 9, further comprising:program instructions, stored on at least one of the one or morecomputer-readable tangible storage devices for execution by at least oneof the one or more processors via at least one of the one or morecomputer-readable memories, to, before monitoring performance of theescalation in the production environment: query parameters associatedwith the escalation; determine from the parameters that the escalationgenerated a number of errors over a period of time at a rate thatexceeds a threshold; and add the escalation to a list of escalations forperformance monitoring.
 15. The system of claim 9, further comprising:program instructions, stored on at least one of the one or morecomputer-readable tangible storage devices for execution by at least oneof the one or more processors via at least one of the one or morecomputer-readable memories, to gather load data for use in identifyingthe characteristic.
 16. The system of claim 9, further comprising:program instructions, stored on at least one of the one or morecomputer-readable tangible storage devices for execution by at least oneof the one or more processors via at least one of the one or morecomputer-readable memories, to, after deploying the one or morerecommended escalations into the production environment, determine thata defined action of a recommended escalation of the one or morerecommended escalations did not execute, and in response, to notify auser that the defined action of the recommended escalation did notexecute.
 17. A computer program product for improving escalationperformance, the computer program product comprising: one or morecomputer-readable tangible storage devices; program instructions, storedon at least one of the one or more computer-readable tangible storagedevices, to monitor performance of an escalation in a productionenvironment; program instructions, stored on at least one of the one ormore computer-readable tangible storage devices, to identify acharacteristic of the escalation based on the monitored performance;program instructions, stored on at least one of the one or morecomputer-readable tangible storage devices, to create a recommendationfor improving future escalation performance based on the characteristic;program instructions, stored on at least one of the one or morecomputer-readable tangible storage devices, to apply the recommendationto the escalation to form one or more recommended escalations inresponse to an approval of the recommendation; and program instructions,stored on at least one of the one or more computer-readable tangiblestorage devices, to deploy the one or more recommended escalations intothe production environment.
 18. The computer program product of claim17, further comprising: program instructions, stored on at least one ofthe one or more computer-readable tangible storage devices, to monitorperformance of the one or more recommended escalations in a testenvironment before deploying the one or more recommended escalationsinto the production environment; wherein the program instructions todeploy the one or more recommended escalations into the productionenvironment deploy the one or more recommended escalations into theproduction environment in response to a determination that the monitoredperformance of the one or more recommended escalations in the testenvironment is an improvement over the monitored performance of theescalation in the production environment.
 19. The computer programproduct of claim 17, wherein: the program instructions to identify thecharacteristic of the escalation based on the monitored performance:read a log data structure associated with the escalation; determine fromthe log data structure that the escalation is associated with first andsecond customers; and identify in the log data structure a number ofdatabase transactions performed by the escalation for the firstcustomer; wherein the characteristic is that that the number of databasetransactions performed by the escalation for the first customer exceedsa threshold; and wherein the recommendation comprises: disassociatingthe first customer from the escalation; splitting the escalation intofirst and second escalations; and associating the second escalation withthe first customer.
 20. The computer program product of claim 17,wherein: the characteristic is that one or more durations of executionof the escalation vary from an expected duration of execution of theescalation by a threshold, and the recommendation comprises adjusting anexecution time of the escalation.
 21. The computer program product ofclaim 17, wherein: the characteristic is that the escalationhistorically processes records in a database at a rate that is slowerthan a rate at which another escalation processes records in thedatabase, and the recommendation comprises executing the escalation at afrequency that is higher than a frequency at which the escalation isconfigured to execute.
 22. The computer program product of claim 17,further comprising: program instructions, stored on at least one of theone or more computer-readable tangible storage devices, to, beforemonitoring performance of the escalation in the production environment:query parameters associated with the escalation; determine from theparameters that the escalation generated a number of errors over aperiod of time at a rate that exceeds a threshold; and add theescalation to a list of escalations for performance monitoring.
 23. Thecomputer program product of claim 17, further comprising: programinstructions, stored on at least one of the one or morecomputer-readable tangible storage devices, to gather load data for usein identifying the characteristic.
 24. The computer program product ofclaim 17, further comprising: program instructions, stored on at leastone of the one or more computer-readable tangible storage devices, to,after deploying the one or more recommended escalations into theproduction environment, determine that a defined action of a recommendedescalation of the one or more recommended escalations did not execute,and in response, to notify a user that the defined action of therecommended escalation did not execute.